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Testing  to  Ensure  Compliance 
with  1%  UXO  Limitations 


Abstract 

US  law  regarding  cluster  munitions  prohibits  the  sale  or  transfer  of  technology,  export 
license,  or  military  assistance  unless  the  submunitions  do  not  result  in  more  than  1% 
unexploded  ordnance  (UXO)  across  the  range  of  intended  operational  environments. 
Multiple  US  allies  want  to  upgrade  their  capabilities  by  licensing  or  otherwise  using  US 
technology  to  improve  the  perfonnance  of  their  cluster  munitions  systems.  Testing  is 
required  to  establish  compliance  with  the  1%  limit  on  UXO.  Current  Acquisition, 
Technology,  and  Logistics  (AT&L)  guidance  requires  that  “a  tailored  test  and  evaluation 
approach  is  necessary  to  detennine,  with  reasonable  confidence,  compliance  with  the 
requirement.”  This  paper  describes  some  historical  approaches  to  developmental  and 
operational  testing  in  which  one  of  the  goals  was  munition  reliability  and  demonstrates 
how  variations  of  these  approaches  might  be  applied  to  meeting  legal  requirements  in  the 
UXO  example.  Related  analyses  isolate  some  of  the  pitfalls  and  risks  inherent  in  testing 
and  attempt  to  provide  suggestions  for  best  practices  to  support  statute  and  guidance. 

Introduction 

An  allied  nation  had  a  manufacturing  license  agreement  (MLA)  granted  by  the 
Department  of  State  to  produce  rocket  systems  that  included  M77  submunitions  (see 
Figure  1).  The  MLA  expired  in  201 1.  The  ally  desires  to  continue  production  of  the 
submunition,  using  an  indigenously  developed  self-destruct  fuze  (SDF)  that  will  reduce 
the  UXO  rate  to  less  than  1%,  as  required  by  US  law.  The  Office  of  the  Secretary  of 
Defense  (OSD)  interprets  the  language  in  the  law  (...  the  submunitions  ...do  not  result  in 
more  than  1%  unexploded  ordnance  across  the  range  of  intended  operational 
environments)  to  require  that  the  1%  threshold  has  to  be  met  under  all  operating 
conditions. 

Preliminary  testing  was  conducted  and  assessed  by  the  US  Anny  in  spring  2013  for 
various  ranges  and  temperatures  to  explore  perfonnance  over  the  operational 
environment.  During  this  testing,  the  1%  threshold  was  met  under  some  conditions  but 
not  others.  The  testing  provided  information  on  how  the  munitions  should  be  employed 
and  how  they  should  be  tested  to  ensure  compliance  with  statute. 

The  ally  has  proposed  a  more  extensive  modified  test  to  establish  UXO  rates  more 
precisely.  This  paper  describes  the  work  done  by  Tactical  Warfare  Systems  (TWS)  and 
Developmental  Test  &  Evaluation  (DT&E)  to  assess  the  adequacy  of  the  proposed  test 
and  determine  passing  criteria. 
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Figure  1.  M77  Submunition 


Cluster  Bomb  Unit  (CBU)-105:  A  Precedent  for  Foreign  Military  Sales 
(FMS)  of  a  Submunition  System 

The  CBU-105  Sensor  Fuzed  Weapon  (SFW)  has  been  in  development  since  the 
early  1990s,  with  several  upgrades.  The  SFW  dispenses  10  submunitions,  each  with 
4  skeet  warheads.  See  Figure  2.  Over  that  time,  the  tested  UXO  rate  on  these  warheads 
has  dropped  by  an  order  of  magnitude  of  around  5%  to  around  0.5%.  The  latest  variant 
underwent  17  separate  tests — primarily  production  verification  tests — from  2002  until 
2008.  These  tests  yielded  3  duds  classified  as  UXO  out  of  625  armed  warheads.  This 
testing,  in  conjunction  with  engineering  analyses,  was  viewed  as  sufficient  to  determine 
compliance  with  the  1%  UXO  requirement.  Retrospectively,  we  analyzed  the  combined 
data  from  these  tests  and  found  that  a  UXO  rate  of  less  than  1%  was  demonstrated  with 
87%  confidence. 
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Initial  Testing  of  the  SDF  and  Lessons  Learned 

The  SDF  was  tested  at  the  White  Sands  Missile  Range  (WSMR)  in  spring  2013, 
using  rockets  carrying  inert  grenades.  Three  missions  (four  rockets  each)  were  fired,  one 
each  at  long  (43.8),  medium  (34.6  km),  and  short  (16.8  km)  range.  In  addition,  the 
potential  impact  of  temperature  conditioning  was  explored  by  conditioning  at  cold, 
ambient,  and  hot  conditions.  The  effects  of  the  range  to  target  and  the  temperature  of  the 
munitions  are  routinely  measured  during  the  testing  of  munitions  systems  to  ensure 
functionality  across  the  operating  environment. 

Range-safety  considerations  constrained  the  short-range  shots  to  a  single  impact 
point,  precluding  separation  by  temperature  at  that  range.  Further,  two  medium-range 
shots  (one  hot  and  one  ambient)  impacted  too  close  together  to  detennine  the  carrier  of 
the  submunitions.  This  situation  confounded  attempts  to  separate  range  and  temperature 
effects. 

The  fuzes  were  the  system  under  test,  so  inert  grenades  were  used  to  simplify  test 
conduct  and  scoring.  Using  inert  grenades  also  eased  the  safety  constraints,  especially  for 
clearance.  However,  premature  detonation  of  a  grenade  in  the  air  can  cause  the  fuze 
failure  of  neighboring  grenades,  so  potentially  important  failure  modes  were  excluded. 
Despite  these  limitations,  the  resulting  data  (see  Table  1)  provided  important  insights. 


Table  1.  SDF  Data  from  2013  WSMR  Testing 


Mission 

Range 

Condition 

Rockets 

Loaded 

Submunitions 

Recovered 

Submunitions 

Total 
Number 
of  UXO 

SAT-1 

43.8  km 

Cold 

1 

504 

503 

0 

43.8  km 

Cold 

1 

504 

487 

0 

43.8  km 

Ambient 

1 

504 

493 

7 

43.8  km 

Ambient 

1 

504 

504 

6 

SAT-2 

34.6  km 

Hot 

1 

504 

498 

1 

34.6  km 

Hot 

1 

504 

993 

9 

34.6  km 

Ambient 

1 

504 

34.6  km 

Ambient 

1 

504 

498 

0 

SAT-3 

16.8  km 

Hot 

1 

504 

1,993 

38 

16.8  km 

Cold 

1 

504 

16.8  km 

Ambient 

1 

504 

16.8  km 

Ambient 

1 

504 

Averaged  over  all  shots,  the  system  UXO  rate  is  1.02%,  which  barely  exceeds  the 
allowed  limit.  The  short-range  shots  are  the  primary  contributor  to  failures,  with  a  nearly 
2%  UXO  rate  (three  or  four  times  that  of  the  shots  at  medium  and  long  range).  However, 
the  ally’s  Operational  Mission  Profile  (OMP)  calls  for  infrequent  (7.5%)  shots  at  short 
range,  with  the  bulk  at  intennediate  range  (70%).  Other  systems  can  engage  targets  at  the 
shorter  ranges,  so  the  current  proposal  is  to  restrict  usage  to  above  25  km.  The  maximum 
range  will  also  be  reduced  somewhat.  Rocket  velocity  at  the  time  of  dispersal  is  believed 
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to  be  a  contributor  to  fuze  failure,  and  these  changes  eliminate  the  higher  velocity 
dispersals,  as  will  be  discussed  in  the  next  section. 

The  impact  of  temperature  on  performance  is  less  clear.  Overall,  UXO  rates  for  the 
missions  at  medium  and  at  long  range  were  below  1%.  However,  the  long-range  rockets 
at  ambient  temperature  had  13  failures  among  997  submunitions — a  1.3%  failure  rate 
(higher  than  the  1%  threshold  with  80%  confidence).  Long-range  rockets  at  cold 
temperature  showed  no  failures  (lower  than  the  1%  threshold  with  99.99%  confidence). 
This  performance  indicates  that  the  temperature  does  matter  and  that  the  rockets  must  be 
tested  at  various  temperatures  to  ensure  functionality  “across  the  operating  environment.” 

At  medium  range,  there  is  no  apparent  effect  due  to  temperature,  although  this 
finding  is  obscured  by  the  inability  to  separate  submunitions  from  two  of  the  shots.  (Note 
that  the  merged  data  from  the  hot  and  ambient  medium-range  shots  that  were  separately 
scored  ( 1  failure  in  996)  do  not  appear  consistent  with  the  data  from  the  rockets  that 
impacted  together  (9  failures  in  993  shots)).  At  short  range,  there  is  no  ability  to  tease  out 
temperature  dependence,  since  the  rockets  all  impacted  in  the  same  area. 

Two  conclusions — applicable  to  the  employment  and  the  testing  of  this  system, 
respectively — can  be  drawn  from  these  data.  First,  it  can  be  concluded  with  99.9+% 
confidence  that  the  UXO  rate  on  short  shots  exceeds  1%.  Short-range  use  will  need  to  be 
restricted.  Second,  it  can  be  concluded  from  the  long-range  shots  that  temperature  is  a 
factor  that  must  be  considered. 

These  results  led  to  a  proposal  from  our  ally  for  an  extensive  test  and  a  reduced 
employment  envelope  (longer  minimum  range  and  shorter  maximum  range)  for  use  of  the 
munitions.  The  details  of  this  proposal  and  the  considerations  in  determining  the  passing 
criteria  are  discussed  in  the  next  section. 

Proposed  Extensive  Testing  of  SDF 

The  basic  steps  in  putting  this  test  together  are  as  follows: 

•  Identify  the  operational  factors  to  be  considered, 

•  Specify  the  extent  of  the  test,  and 

•  Decide  on  the  “passing”  criteria. 

The  factors  that  are  considered  in  these  tests  to  determine  effectiveness  are  the 
temperature  conditioning  of  the  round  and  the  range  to  target.  Temperature  conditioning 
is  a  standard  feature  of  munitions  testing,  and  evidence  from  the  2013  testing  suggests 
that  the  temperature  does  matter  for  the  longest  range  shots.  Thus,  hot,  cold,  and  ambient 
shots  will  be  included. 

The  evidence  from  the  2013  testing  for  an  effect  due  to  range  is  also  compelling. 

The  short-range  shots  clearly  produce  more  UXO  (38/1,993)  than  either  medium-range 
(10/1,989)  or  long-range  (13/1,987)  shots.  The  superiority  of  the  medium-  range  shots  to 
the  long-range  shots  is  statistically  marginal.  The  UXO  rate  correlates  to  the  speed  of  the 
rocket  at  the  time  of  expulsion,  which  is  shown  in  Figure  3  as  a  function  of  range  to 
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Figure  3.  Rocket  Velocity  at  Warhead  Expulsion 


target.  In  Figure  3,  the  dashed  lines  topped  by  an  arrow  indicate  the  2013  test  ranges,  and 
the  dashed  lines  topped  by  a  square  indicate  the  proposed  ranges  for  the  future  test. 

For  artillery-shell-delivered  submunitions,  damage  by  collision  following  expulsion 
has  been  identified  as  a  source  of  fuze  failure. 1  The  combination  of  successful  earlier 
medium-range  shots,  along  with  a  credible  engineering  explanation  of  why  these  shots 
perfonned  best,  suggests  that  testing  at  the  medium  ranges  in  addition  to  the  proposed 
short-  and  long-range  testing  is  unnecessary.  As  mentioned  previously,  the  proposed 
minimum  and  maximum  ranges  for  employment  of  the  rockets  are  adjusted  so  that  the 
submunition  expulsion  velocity  is  decreased.  Based  on  the  test  results  and  engineering 
assessments,  the  minimum  range  for  the  rockets  is  increased  from  16.8  to  26  km,  and  the 
maximum  range  is  decreased  from  43.8  to  42  km.  Intended  use  of  the  system  is  at  ranges 
of  30  km  and  beyond.  The  OMP  is  thus  modified  to  have  a  26-km  minimum  range  (one 
of  the  ranges  to  be  tested).  Therefore,  we  have  six  “bins”  (two  ranges  by  three 
temperatures)  to  test,  in  contrast  to  the  2013  test,  which  had  nine  potential  conditions 
(three  ranges  by  three  temperatures,  five  of  which  were  distinguished  in  the  testing). 

These  operational  factors  are  to  be  explored  in  a  much  more  extensive  test  in  which 
54  rockets  rather  than  12  rockets  will  be  used.  The  increased  number  of  rockets,  coupled 
with  the  decreased  number  of  bins  to  explore,  leads  to  a  more  precise  determination  of 
UXO  rate  than  was  possible  in  the  previous  test. 


Another  possible  source  of  fuze  failure  is  damage  to  the  fuze  of  one  submunition  when  a  different 
submunition  detonates  on  dispersal,  which  is  why  live  submunitions  are  required  for  the  proposed  test. 
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Many  tenninologies  are  used  for  discussing  the  precision  of  a  determination  of  this 
type.  The  testing  process  is  a  sampling  of  a  large  population  to  detennine  one  or  more 
characteristics.  The  larger  the  sample,  the  more  precise  the  determination.  In  this  case, 
the  intent  is  to  determine  whether  a  system  meets  a  threshold  for  performance  (less  than 
1%  UXO).  There  is  a  “true”  value  of  the  UXO  for  the  entire  population  of  submunitions 
and  an  “observed”  or  “tested”  value.  The  hope  is  that  the  true  and  observed  values  will 
either  both  pass  or  both  fail  the  threshold;  however,  for  any  size  test,  there  will  be  some 
possibility  of  error  of  either  of  two  types: 

•  The  system  meets  spec  but  fails  the  test  (“producer  risk”),  or 

•  The  system  fails  to  meet  spec  but  passes  the  test  (“consumer  risk”). 

Expanding  the  scope  of  the  test  reduces  the  probability  of  error  of  either  type,  no  matter 
what  the  true  performance  value  is. 

In  addition  to  reducing  the  probability  of  an  error,  the  “risk”  can  be  shifted  from  the 
consumer  to  the  producer  by  imposing  a  requirement  that  test  results  establish  a 
confidence  of  greater  than  50%  that  the  system  has  met  or  exceeded  the  performance 
requirement.  In  general,  this  condition  means  requiring  perfonnance  in  the  test  that  is 
better  than  the  point  estimate  associated  with  the  requirement. 

For  systems  that  just  barely  miss  or  just  barely  meet  spec,  the  consumer’s  risk  is 
approximately  one  minus  the  required  confidence  level,  and  the  producer’s  risk  is 
approximately  the  required  confidence  level.  As  the  true  performance  exceeds  or  misses 
the  required  performance  by  greater  amounts,  the  risk  declines  since  it  becomes  more  and 
more  unlikely  that  the  test  results  will  lead  to  an  incorrect  conclusion. 

This  use  of  “risk”  is  essentially  as  a  probability  and  differs  from  the  acquisition  use 
in  which  risk  is  a  probability  associated  with  a  consequence.  If  risk  is  thought  about  as 
incorporating  the  severity  of  the  consequence,  the  advantage  of  a  more  extensive  test  is 
not  just  that  the  probability  of  error  decreases,  but  also  that  the  probability  of  a  large  error 
collapses.  This  concept  is  illustrated  later  in  the  paper. 

With  54  rockets  (each  carrying  504  submunitions)  and  six  categories  for  the 
temperature  and  range  factors,  there  are  9  rockets  (4,536  submunitions)  in  each  bin.  Thus, 
45  or  fewer  unexploded  submunitions  in  each  bin  would  establish  a  point  estimate  for 
each  condition  tested  of  1%  or  less  for  the  UXO.  Since  the  measurements  are  randomly 
sampling  a  population,  the  point  estimate  is  the  value  for  which  there  is  an  approximately 
50-50  chance  of  the  entire  population  being  either  above  or  below  the  estimate.  Requiring 
a  more  stringent  threshold  gives  measurable  confidence  that  the  UXO  rate  is  below  1%. 
Table  2  lists  the  confidence  levels  under  consideration  and  the  maximum  number  of 
UXO  allowed. 

Returning  to  the  discussion  of  consumer’s  and  producer’s  risk  with  a  quantitative 
example  of  how  consumer’s  risk  “collapses,”  Table  3  compares  the  9-rocket  case  to  a 
1+  rocket  case.  (For  the  1+  rockets  case,  the  number  of  submunitions  was  adjusted  so  the 
confidence  levels  can  be  matched  at  integer  thresholds.) 
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Table  2.  Confidence  Levels  under  Consideration  and  Maximum  Number  of  UXO  Allowed 


Nominal  Confidence 
Level  Required 

Maximum  Number  of 
UXO  from  9  Rockets 
(4,536  Submunitions) 

Confidence  Level 

* 

Established 

Point  estimate 

45 

48.2% 

80% 

39 

80.8% 

90% 

36 

91 .0% 

*  This  is  the  probability  of  exceeding  the  “Maximum  Number’’  if  the  actual  UXO  rate  were  1%. 


Table  3.  Consumer’s  Risk  for  Two  Tests  with  80%  Confidence  Thresholds 


Number  of 
Submunitions 

Confidence  Level/ 
UXO  Threshold 

Consumer’s  Risk 

If  True  UXO  Rate 

Is  1.01% 

Consumer’s  Risk 

If  True  UXO  Rate 

Is  1.25% 

557  (1+  rockets) 

80.8%/3  UXO 

18.6% 

8.2% 

4,536  (9  rockets) 

80.8%/39  UXO 

17.5% 

0.80% 

The  consumer’s  risk  when  the  UXO  rate  is  1.01%  (1%  above  spec)  is  about  the 
same  in  both  cases  but  becomes  negligible  for  the  9-rocket  case  if  the  UXO  rate  rises  to 
1.25%  (the  spec  is  missed  by  25%). 

It  was  noted  previously  that  requiring  confidence  levels  showing  that  the  spec  has 
been  met  is  how  the  consumer  can  reduce  his  risk,  but  the  risk  is  then  shifted  to  the 
producer.  The  producer  reduces  risk  by  designing  margin  into  the  system.  Again,  an 
extensive  test  helps.  If  the  producer  successfully  executes  a  design  for  0.75%  fuze  failure, 
there  is  still  a  60%  probability  that  the  system  will  fail  a  1 -rocket  test  with  an  80% 
confidence  requirement.  However,  this  probability  of  failure  drops  to  only  17%  in  the 
9-rocket  test,  again  with  the  80%  confidence  required. 

Conclusion 

The  proposal  for  a  54-rocket  test  explores  the  effects  of  temperature  conditioning 
and  range  on  the  UXO  rate  for  the  submunitions.  This  extensive  testing  reduces  the 
probability  of  sampling  uncertainty,  leading  to  an  erroneous  conclusion.  In  addition,  the 
use  of  an  80%  confidence  requirement  on  this  extensive  test  essentially  eliminates  any 
possibility  of  passing  the  fuze  if  the  failure  rate  is  appreciably  about  the  1%  threshold. 
There  is  no  prospect  for  large  errors  of  this  type.  Further,  because  of  the  extent  of  the  test, 
a  reasonable  design  margin  allows  the  producer  to  reduce  the  risk  of  an  erroneous  adverse 
result  to  a  tolerable  level. 

Abbreviations 

CBU  Cluster  Bomb  Unit 

DT&E  Developmental  Test  &  Evaluation 

FMS  Foreign  Military  Sales 

MLA  manufacturing  license  agreement 

OMP  Operational  Mission  Profile 

OSD  Office  of  the  Secretary  of  Defense 
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SDF 

self-destruct  fuze 

SFW 

Sensor  Fuzed  Weapon 

TWS 

Tactical  Warfare  Systems 

UXO 

unexploded  ordnance 

WSMR 

White  Sands  Missile  Range 
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